Learning, Moving, And Predicting With Global Motion Representations

In order to effectively respond to and influence the world they inhabit, animals and other intelligent agents must understand and predict the state of the world and its dynamics. An agent that can characterize how the world moves is better equipped to engage it. Current methods of motion computation rely on local representations of motion (such as optical flow) or simple, rigid global representations (such as camera motion). These methods are useful, but they are difficult to estimate reliably and limited in their applicability to real-world settings, where agents frequently must reason about complex, highly nonrigid motion over long time horizons. In this dissertation, I present methods developed with the goal of building more flexible and powerful notions of motion needed by agents facing the challenges of a dynamic, nonrigid world. This work is organized around a view of motion as a global phenomenon that is not adequately addressed by local or low-level descriptions, but that is best understood when analyzed at the level of whole images and scenes. I develop methods to: (i) robustly estimate camera motion from noisy optical flow estimates by exploiting the global, statistical relationship between the optical flow field and camera motion under projective geometry; (ii) learn representations of visual motion directly from unlabeled image sequences using learning rules derived from a formulation of image transformation in terms of its group properties; (iii) predict future frames of a video by learning a joint representation of the instantaneous state of the visual world and its motion, using a view of motion as transformations of world state. I situate this work in the broader context of ongoing computational and biological investigations into the problem of estimating motion for intelligent perception and action

Direct Control of Visual Perception with Phase-specific Modulation of Posterior Parietal Cortex

We examined the causal relationship between the phase of alpha oscillations (9–12 Hz) and conscious visual perception using rhythmic TMS (rTMS) while simultaneously recording EEG activity. rTMS of posterior parietal cortex at an alpha frequency (10 Hz), but not occipital or sham rTMS, both entrained the phase of subsequent alpha oscillatory activity and produced a phase-dependent change on subsequent visual perception, with lower discrimination accuracy for targets presented at one phase of the alpha oscillatory waveform than for targets presented at the opposite phase. By extrinsically manipulating the phase of alpha before stimulus presentation, we provide direct evidence that the neural circuitry in the parietal cortex involved with generating alpha oscillations plays a causal role in determining whether or not a visual stimulus is successfully perceived

Population response magnitude variation in inferotemporal cortex predicts image memorability

Most accounts of image and object encoding in inferotemporal cortex (IT) focus on the distinct patterns of spikes that different images evoke across the IT population. By analyzing data collected from IT as monkeys performed a visual memory task, we demonstrate that variation in a complementary coding scheme, the magnitude of the population response, can largely account for how well images will be remembered. To investigate the origin of IT image memorability modulation, we probed convolutional neural network models trained to categorize objects. We found that, like the brain, different natural images evoked different magnitude responses from these networks, and in higher layers, larger magnitude responses were correlated with the images that humans and monkeys find most memorable. Together, these results suggest that variation in IT population response magnitude is a natural consequence of the optimizations required for visual processing, and that this variation has consequences for visual memory